This sounds fascinating. I'd love to hear critiques of it from people in the field. Is it a good study?

Also the tweets are from 2009-10. I wonder what would happen with messages from 2016-present.
posted by medusa at 10:33 AM on January 9, 2022 [5 favorites]

but I'm not on Twitter
posted by philip-random at 10:41 AM on January 9, 2022 [3 favorites]

Given that the typical Twitter user is younger (median age = 31 years; Fox, Zickurh, & Smith, 2009) than the typical person at risk for AHD, it is not obvious why Twitter language should track heart-disease mortality. The people tweeting are not the people dying. However, the tweets of younger adults may disclose characteristics of their community, reflecting a shared economic, physical, and psychological environment. At the individual level, psychological variables and heart-disease risk are connected through multiple pathways, including health behaviors, social relationships, situation selection, and physiological reactivity (Friedman & Kern, 2014). These pathways occur within a broader social context that directly and indirectly influences an individual's life experiences. Local communities create physical and social environments that influence the behaviors, stress experiences, and health of their residents (Diez Roux & Mair, 2010; Lochner, Kawachi, Brennan, & Buka, 2003). Epidemiological studies have found that the aggregated characteristics of communities, such as social cohesion and social capital, account for a significant portion of variation in health outcomes, independently of individual-level characteristics (Leyland, 2005; Riva, Gauvin, & Barnett, 2007), such that the combined psychological character of the community is more informative for predicting risk than are the self-reports of any one individual. The language of Twitter may be a window into the aggregated and powerful effects of the community context.

This was interesting.
posted by medusa at 10:43 AM on January 9, 2022 [11 favorites]

Is it a good study?

No. It relies on twitter users who include their geographic information, which is an unrepresentative 3% sample of all twitter users.

The way they compare the predictive power of two models is bad. You can't just include/exclude a variable because random noise affects prediction, you need to permute the variable you are testing.


"This resulted in 148 million countymapped tweets across 1,347 counties." Why only 1,347? Theres over 3000 counties in the US. What happened to the rest and how reliable is that?

Later we see: " Counties for which reliable CDC or Twitter language data were unavailable are shown in white." What?

I mean, this doesn't pass the smell test for me.
posted by MisantropicPainforest at 10:45 AM on January 9, 2022 [7 favorites]

This seems way too good to be true. All of the socioeconomic and health risk factors for heart disease mortality that we've studied in detail over many years are worse than some nebulous word negativity data from Twitter? That seems pretty questionable.

Here is a critique that finds the opposite association with suicide deaths (more negative words, less suicide), which seems odd. There are a number of other issues identified in this analysis.
posted by ssg at 10:47 AM on January 9, 2022 [1 favorite]

Also, I think we need take to take public health results from psychologists with a very large grain of salt. We've seen so much bad research and so many results that can't be reproduced in psychology that assuming any kind of strong claim like this is false until proven otherwise is pretty justifiable.
posted by ssg at 10:53 AM on January 9, 2022 [6 favorites]

Why only 1,347? Theres over 3000 counties in the US. What happened to the rest and how reliable is that?
They say they picked high-density counties. In a county with a low population, just a couple Twitter users could skew the results.
posted by CheeseDigestsAll at 10:54 AM on January 9, 2022 [1 favorite]

Oh, well. It would have been interesting if living with obnoxious younger people causes heart disease, but the real world isn't that simple.
posted by Nancy Lebovitz at 11:08 AM on January 9, 2022

"They say they picked high-density counties. In a county with a low population, just a couple Twitter users could skew the results." This screams of "lets cut a bunch of observations until we get what we want"

So this only applies for a non-random sample of US counties based on a non-random sample of twitter users. Cool.
posted by MisantropicPainforest at 11:10 AM on January 9, 2022 [1 favorite]

It relies on twitter users who include their geographic information, which is an unrepresentative 3% sample of all twitter users.

We already know that Twitter users are unrepresentative of the general population, anyway. Change the title of the paper to say "Geolocated Twitter Users" instead of "Twitter" if you want to. It's still a correlation between one set of data and another.

And it seems very reasonable to exclude counties where there isn't sufficient data, either from Twitter or the CDC. Does that bias things? Probably! Worth looking into.

I mean, in the end their claim is basically "Hey, this data over here correlates with that data over there. That's kinda interesting! But yeah we don't know why." Nobody should go off and try to make predictions using this, let alone policy. But as I see it the researchers looked in a place, found something interesting there, and are in effect suggesting other people should look there too. If other people find it isn't real or can be explained by some third, more relevant factor or whatever else, great!
posted by whatnotever at 11:10 AM on January 9, 2022 [15 favorites]

That's exactly what they are doing and its an atrocious way to do science!
posted by MisantropicPainforest at 11:19 AM on January 9, 2022 [2 favorites]

The science may not be 100% sound. But - honestly, if it catches the attention of enough of the people with power in a given community and makes them say "hmm, maybe we better pay a little closer attention to Twitter overall and watch what people in our town are saying", then - that is probably a net win, no matter WHAT those people notice.

Maybe they're convinced that "yeah maybe we should spend a little more money on heart disease awareness" - but also, maybe they notice "WTF, there's a dude with a Proud Boys chapter in town? Since when?" or "oh crap, there seems to be a lot of people talking about broken streetlights" or "huh, a lot of people are complaining about not enough hours for library access, that's an easy fix". It may not be the thing they were looking for, but more eyes on the local buzz can only be good, yeah?
posted by EmpressCallipygos at 11:50 AM on January 9, 2022 [1 favorite]

Reminds me of the "Yankee Candle complaints predict Covid outbreaks" thing. Kinda interesting; mostly silly.
posted by EatTheWeek at 12:11 PM on January 9, 2022 [3 favorites]

For anyone who has actually read the study... when they say "predicts" do they actually mean prediction, or is it more like those Gottman marriage predictions where they're talking about how many data points that were used to create the model fit the model they created? Like... did they hold some data back and do predictions on it after they'd made their model, or are they just telling us how overfitted their model is?

In the event that the study is true, I sometimes wonder whether people act miserable and mean because they're sick, rather than that they get sick because they act miserable and mean. It was my "mean auntie" who made me think of this, after finding out after her death that she had suffered from a bunch of chronic conditions. Obviously some people can smile through all the pain, but I wonder if on average anger is a result of heart disease rather than a cause.
posted by clawsoon at 12:33 PM on January 9, 2022 [2 favorites]

I suppose the US census has more power, but is less sexy. In the end, why wouldn't you just use the census data? Do risks change faster than the ACS data? doubtful.

I long for the day when we have county-level air pollution data to discuss, since that is shown to cause heart-disease related mortality. When the federal government stumbles across such data, as with the BP disaster, the correlations are simple enough, and the mechanisms known.

Perhaps the impacts of air pollution are too well documented? And that's why we can't look for them?

One big problem is that air pollution comes from the political donor class.

We can't even get child care, or cheap gas, if that would hurt the Coal Senators' financial prospects--so not a surprise that the Fossil Fuel Senators would stop us collecting evidence that would hurt their bottom line directly by saving american hearts.

Also, guess what, air pollution, access to electricity and internet (and thus => air pollution data collection techniques) and access to health care are all likely going to be correlated to those nebulous 'social cohesion' factors discussed by the authors--Social Determinants of Health are also going to be Social Determinants of Scientific Research and Social Determinants of Broadband Access.

So, good luck getting the government to fund testing the air to protect our hearts; especially in areas without political power; when we can barely get them to do COVID testing.

We are better off inventing other ways to improve heart health. Maybe if we correlate around the bush long enough, we could catch them sleeping.

So, to me, in light of what is known about air pollution, and the degree to which we know what we need to learn, and refuse to learn it, this study searching for new mechanisms is all very

"Let them correlate Twitter!"
posted by eustatic at 1:00 PM on January 9, 2022 [3 favorites]

They fitted their model on 90% of counties and tested it on the remaining 10%.

The study question appears to be along the lines of "does anger cause heart disease?" I thought that mentality went out with "does stress cause ulcers?"

"compared with the counties in the final sample, the excluded counties had smaller populations, higher rates of [heart disease], lower income, and lower levels of education." (footnote 1; I removed the statistical terms for readability here)

This seems like the most cherry-picked example of bad correlation here. On the level of pirates preventing jclimate change. At least the Flying Spaghetti Monster is satire. I don't even know what this paper is, but science is ain't.
posted by basalganglia at 1:08 PM on January 9, 2022 [3 favorites]

Or as my research mentor used to say (more apt here than ever): Garbage in, garbage out.
posted by basalganglia at 1:27 PM on January 9, 2022

If I understand the paper correctly from a skim, they are saying that, from your language on twitter, they can predict if you live in a county that has high cardiovascular disease rates. This conclusion would be credible and fairly uninteresting.

It's like saying I can predict if you'll be hit by lightning based on your Instagram, which seems like a magic trick until you realize by my metrics "success" just means I've figured out out if you live an area that gets lots of thunderstorms.
posted by mark k at 2:43 PM on January 9, 2022 [4 favorites]

right. we already know that zip codes are destiny. what this study shows is that, sometimes twitter lets you see people's zip codes.
posted by eustatic at 2:54 PM on January 9, 2022 [2 favorites]

BRB whipping a Psychology Simulator called "Replication Crysis".
posted by symbioid at 3:21 PM on January 9, 2022 [2 favorites]

Oh, huh, this was a while ago. I was in this lab at the time. tl;dr: I believe the effect, but I don’t believe there’s necessarily anything that interesting or actionable behind it.

The effects are statistically robust, controlling for everything from age/gender/race/income demographics to population to density. Basically, language picks up a significant portion of what the census misses. That’s the bit many people here are missing.

Something about language above and beyond that stuff is out-of-sample predictive of heart disease. A lot of the stories around why it’s negative language or whatever are a little fishy. Really the language is reflective of some aspect of the culture that remains when you residualize out everything that the census captures.

sometimes twitter lets you see people's zip codes

Nnnnnno… it lets you guess better-than-chance about heart disease (or a lot of things, really) from that county. It’s not overfit, and there are enough counties in the study that the effect isn’t driven by a few of them.

Dropping counties with low Twitter activity and/or low population + low raw disease numbers makes the predictions more robust when you’re weighting your observations the same.
posted by supercres at 8:30 PM on January 9, 2022 [6 favorites]

(Last thing: will note that the research group is PI’d by Marty Seligman, who’s a real piece of shit, but he has very little bearing on anything that happens day-to-day, probably even less than when I left 5 years ago.)
posted by supercres at 8:44 PM on January 9, 2022 [1 favorite]

I think there is a common cause for mean, grumpy stressed people and heart disease, but that it’s several links back (intermediate causes and effects) along each leg.

When we have a Grand Unified Theory of heart disease, that will be part of it.
posted by jamjam at 11:17 PM on January 9, 2022

if it catches the attention of enough of the people with power in a given community and makes them say "hmm, maybe we better pay a little closer attention to Twitter overall and watch what people in our town are saying", then - that is probably a net win

Maybe this is part of what you're saying, but we've seen Twitter's (IMO negative) effect on journalism and the funding of investigative journalism over the last decade. Or like, the rise and fall of the Arab Spring liberatory social media narrative. If paying more attention to Twitter is easier and cheaper than (and doesn't require the rigorous IRB approval of) more scientific studies/interventions, who's to say e.g. Ron Johnson doesn't just slash state funding of U of Wisconsin's medical programs and throw the money at a couple of brogrammers with a random walk algorithm and the Twitter API? I know it's something of a slippery slope argument, but . . .
posted by aspersioncast at 10:34 AM on January 10, 2022

The effects are statistically robust, controlling for everything from age/gender/race/income demographics to population to density. Basically, language picks up a significant portion of what the census misses. That’s the bit many people here are missing.

There are so many 'garden of forking path' choices here that I don't actually believe its true. I believe that the sample they curated has robust correlations, but thats it.
posted by MisantropicPainforest at 4:44 PM on January 10, 2022

Yeah unfortunately “desk drawer” significance hacking is a huge glaring potential problem here with a data set that really has to tiptoe around Twitter TOS and therefore can’t be completely open. Having sat in on those lab meetings, and worked with that data set a lot (corpus and outcomes), I don’t think that’s the case, but no one has any good reason to believe my expertise or trust my word.

One of the responses used the fact that the corpus could also predict suicidality as a knock against the paper— “oh you’re just picking up something generic with enough features that it’ll predict anything”. I think it speaks to the power of community language-as-reflection-of-behavior regardless of specific statistic ¯\_(ツ)_/¯. It’s an ongoing conversation that they have with reviewers all the time.

Here’s a reanalysis + non-replication response, WWBP’s response to response with replication, finally another in the thread. I think technically both parties are right in various ways and the distinction is kind of boring.
posted by supercres at 9:05 PM on January 10, 2022

(My indifference is the result of thinking about this stuff too hard for too long, too long ago. I did have to double check whether I was an author on this paper before wading in. Thankfully I’m not. I still think there’s interesting power in community language that differs from individual language in really odd — sometimes opposite direction! — ways.)
posted by supercres at 11:09 PM on January 10, 2022

Thank you, up there, for stating how they knew where the Twitter users were located. I also skimmed through the paper looking for how many individuals made the tweets but so far I can only find the number of tweets; angry but talkative individual users would skew the results, it seems to me. If it does say how many individuals per county were using the hateful language, THAT would at least be useful for figuring out where not to move! (Side note: the Yankee Candle thing seems silly because it was a joke. I made a joke. I regret it now.)
posted by terridrawsstuff at 7:27 AM on January 13, 2022

Alternate hypothesis: It could be that being surrounded by angry people is what gives you heart disease, since this is a county-by-county study rather than an individual-by-individual study.
posted by clawsoon at 7:47 AM on January 13, 2022 [1 favorite]